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Abstract 

The  Navai  Postgraduate  School  is  developing  a  competency  model  for  the  profession  of 
systems  engineering.  There  is  currentiy  no  professional  engineering  occupational  code 
(08XX)  for  systems  engineers,  so  there  is  no  verified  competency  model  source  to  use  for 
human  resource  functions  related  to  systems  engineers.  The  key  objective  of  the  systems 
engineering  competency  career  model  (SECCM)  project  is  to  develop  a  verified  systems 
engineering  career  competency  model  that  can  be  used  as  a  foundation  for  establishing 
construct  validity  for  SE  related  human  resources  actions.  To  ensure  the  SE  competency 
model  will  be  able  to  be  used  this  way,  the  SECCM  project  is  having  0PM  use  a  standard 
approach  to  verify  the  occupational  analysis  in  compliance  with  the  Uniform  Guidelines  on 
Employee  Selection  Procedures  (Uniform  Guidelines).  A  verified  SECCM  provides  a  valuable 
resource  as  a  model  that  can  be  used  for  both  key  human  resource  functions  as  well  as  for 
the  consistent  career  development  of  systems  engineers.  The  SECCM  is  based  on  the  Office 
of  the  Secretary  of  Defense  (OSD)  ENG  Career  Field  Competency  Model,  currently  used  by 
the  ENG  acquisition  community.  The  SECCM  has  enhanced  the  current  ENG  model  through 
the  addition  of  extensive  sets  of  knowledge,  skills,  and  ability  (KSA)  statements  -  each 
defined  in  terms  of  Bloom’s  taxonomy  and  mapped  to  each  of  the  higher  level  system 
engineering  competencies.  Each  competency  and  the  associated  KSAs  are  further 
partitioned  into  a  series  of  typical  career  development  points  (i.e.,  from  entry  level  to  subject 
matter  expert),  to  allow  for  planning  and  tracking  of  a  competency-based  development  of 
systems  engineers.  This  paper  describes  the  development  of  the  SECCM,  and  the  current 
state  of  the  competency  model  and  verification  process. 


Introduction 

There  is  currently  no  professional  engineering  occupational  code  (08XX)  for  systems 
engineers.  These  occupational  series  each  have  a  related  competency  model  that  can  be 
used  for  human  resource  functions.  Since  there  is  no  occupational  series  for  SE,  the  Naval 
Postgraduate  School  (NPS)  is  developing  a  competency  model  for  the  profession  of 
systems  engineering.  The  systems  engineering  competency  career  model  (SECCM)  Is  a 
multi-year  project  funded  by  the  Deputy  Assistant  Secretary  of  the  Navy  (DASN)  for 
Research  Development  Test  and  Evaluation  (RDT&E). 
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Competency  is  defined  as  “an  observable,  measurable  pattern  of  skills,  knowledge, 
abilities,  behaviors  and  other  characteristics  that  an  individual  needs  to  perform  work  roles 
or  occupational  functions  successfully”  (Office  of  Personnel  Management  [0PM]). 
Competency  modeling  is  defined  as  the  activity  of  determining  the  specific  competencies 
that  are  characteristic  of  high  performance  and  success  in  a  given  job  (LaRocca,  n.d.). 
According  to  Joshi  et  al.  (2010),  competency  is  defined  as  the  ability  to  use  the  appropriate 
knowledge,  skills,  and  abilities  (KSAs)  to  successfully  complete  a  specific  job-related  task. 
Systems  engineering  (SE)  competencies  are  defined  based  on  the  knowledge,  skills,  and 
abilities — commonly  referred  to  as  KSA,  KSAs,  KSAA,  or  KSAB — that  are  necessary  for  a 
systems  engineer  to  perform  tasks  related  to  the  discipline.  The  SECCM  will  refer  to  the 
knowledge,  skills,  and  abilities  as  KSAs.  Proficient  systems  engineers  are  expected  to  be 
able  to  integrate,  apply,  and  be  assessed  on  these  KSAs  as  they  develop  competencies 
through  their  education  and  training,  professional  development,  and  on-the-job  experience 
(Khan,  2014). 

To  define  the  attributes  of  a  “good”  competency  model,  we  cite  the  Holt  and  Perry 
(201 1)  guide.  The  Holt  and  Perry  guide  states  that  a  good  competency  model  “goes  through 
many  iterations,  focuses  on  a  specific  aspect  of  competency  yet  is  also  simple  and  easy  to 
understand,  maps  competencies  across  levels,  and  maps  levels  in  a  way  that  is  easy  to 
follow  and  emphasizes  technical  skills.”  In  addition,  a  good  competency  model  serves  as  a 
platform  by  which  individuals  can  assess  their  skill  set  (Holt  &  Perry,  2011). 

Many  organizations  within  the  Department  of  the  Navy  (DoN)  and  Department  of 
Defense  (DoD) — both  the  services  and  4th  Estate — have  SE  competency  models  that  are 
locally  “verified”  or  “validated”  for  their  own  individual  use.  These  uses  include  career 
development,  tracking  education  and  training,  and  understanding  the  work-related  activities 
that  systems  engineers  have  to  accomplish.  These  SE  competency  models  have  been 
verified  or  validated  locally  in  the  sense  that  they  have  proven  useful  in  their  operational 
environment  to  define  what  their  systems  engineers  do.  There  is,  however,  currently  no  SE 
competency  model  verified  in  accordance  with  the  Uniform  Guidelines  on  Employee 
Selection  Procedures  (Uniform  Guidelines;  Biddle  Consulting  Group,  2013).  Only  a  model 
that  is  validated  with  the  Uniform  Guidelines  can  be  used  with  confidence  for  all  human 
resource  (HR)  functions,  especially  for  “high  stakes”  functions  like  hiring,  selection,  writing 
position  descriptions,  and  creating  job  announcements.  Due  to  the  importance  of  having  a 
model  verified  for  HR  functions,  DASN  (RDT&E)  extended  the  invitation  to  sister 
components  and  the  Missile  Defense  Agency  (MDA)  to  participate  in  the  model  verification 
process  in  an  effort  to  make  a  model  that  is  useful  for  all  of  the  DoD. 

The  SECCM  is  based  on  the  Engineer  (ENG)  Career  Field  Competency  Model 
currently  used  by  the  ENG  acquisition  community  to  maintain  consistency  within  the  DoD. 
This  multi-part  ENG  competency  model  addresses  both  core  analytical,  technical  program 
management,  business  acumen,  and  professional  competencies  (Delgado,  2014). 

The  SECCM  has  enhanced  the  current  ENG  model  through  the  addition  of  extensive 
sets  of  KSAs  mapped  to  each  of  the  higher  level  SE  competencies,  defined  over  a  series  of 
typical  career  development  points.  Members  of  the  NPS  SECCM  team  worked  with  the 
National  Defense  Industries  Association  (NDIA)  SE  working  group  (WG)  over  the  past 
several  years  to  help  create  the  initial  ENG  Career  Field  Competency  Model.  The  SECCM 
then  added  KSA  details  from  several  other  existing  systems  engineering  competency 
models,  many  provided  to  the  original  NDIA  SE  WG,  from  a  variety  of  organizations. 

The  main  objective  of  the  SECCM  project  is  to  develop  a  verified  systems 
engineering  career  competency  model  that  can  be  used  as  a  foundation  for  establishing 
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construct  validity  for  SE  related  human  resources  actions.  To  ensure  the  SE  competency 
model  will  be  able  to  be  used  this  way,  the  SECCM  research  team  is  having  the  0PM  use  a 
standard  approach  to  verify  the  occupational  analysis  in  compliance  with  the  Uniform 
Guidelines.  Such  a  verified  SECCM  provides  a  valuable  resource  as  a  model  that  can  be 
used  for  both  key  human  resource  functions  as  well  as  for  the  consistent  career 
development  of  systems  engineers. 

SECCM  Development  Methodology 

Several  competency  models  were  used  to  create  and  iterate  the  SECCM 
development  in  an  effort  to  combine  information  from  existing  sources  to  generate  as 
complete  a  scope  of  SE  KSA  as  possible.  The  ENG  (formerly  SPRDE)  Career  Field 
Competency  Model  was  used  as  a  basis  for  the  set  of  baseline  competencies  for  the 
SECCM.  The  SECCM  development  then  iterated  through  a  series  of  modifications  to  the 
ENG  model  to  a  final  form  developed  through  verification  by  naval  system  command  subject 
matter  experts  (SMEs). 

Existing  Systems  Engineering  Model  Sources 

Sources  for  the  competency  models  used  as  the  baseline  for  the  SECCM  include  the 
International  Council  on  Systems  Engineering  (INCOSE)  United  Kingdom  (UK),  Boeing,  The 
National  Aeronautics  and  Space  Administration  (NASA),  Defense  Acquisition  University 
(DAU)  Systems  Planning,  Research,  Development,  and  Engineering  (SPRDE),  Naval 
Aviation  Systems  Command  (NAVAIR),  MITRE,  Boeing,  Space  and  Naval  Warfare  Systems 
Command  (SPAWAR),  and  the  Naval  Underwater  Warfare  Center  (NUWC)  Newport. 
Collectively,  once  the  various  sets  of  competencies  were  collected  and  organized  by  affinity, 
the  original  SECCM  consisted  of  32  competencies.  Of  these,  29  were  identical  to  the 
original  SPRDE  competencies  (White,  2014).  The  next  step  in  developing  the  original 
SECCM  was  to  identify  any  KSAs  associated  with  these  32  SE  competencies.  Only  three  of 
the  existing  models  (NUWC,  INCOSE  UK,  and  MITRE)  had  significant  levels  of  detail  to 
allow  the  identification  of  KSA.  In  order  to  add  additional  depth  in  the  identification  of  KSA, 
the  DAU  SPRDE  Level  I,  II,  and  III  course  learning  objectives  were  transformed  into  KSAs 
and  added  to  the  SECCM.  Each  entry,  or  KSA,  from  these  sources  was  then  used  to  build 
the  baseline  SECCM. 

The  harmonized  set  of  KSAs  was  analyzed  and  re-organized  based  on  the  affinity 
with  the  respective  competencies.  The  KSAs  from  each  of  the  four  sources  were  re-aligned 
to  fit  the  SECCM  by  first  eliminating  duplication,  then  eliminating  items  that  did  not  seem  to 
be  explicitly  defined  as  a  relevant  SE  competency,  and  lastly  by  re-organizing  all  of  the 
KSAs  into  an  appropriate  competency.  Figure  1  is  an  illustration  of  the  existing  competency 
models  that  were  used  to  create  the  SECCM.  Figure  2  shows  the  models  used  to  derive  the 
SECCM  KSAs. 
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Figure  1.  Competency  Sources  Used  in  the  SECCM 
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Figure  2.  Models  Used  to  Derive  the  SECCM  KSAs 

An  NPS  WG  compared  the  32  SECCM  competencies  with  the  29  in  the  SPRDE 
model.  The  WG  decided  that  two  of  the  three  additional  competencies  were  more  related  to 
contracting,  and  one  was  specific  to  systems  thinking,  so  these  three  competencies  were 
eliminated  as  high  level  competency  categories,  and  the  associated  KSA  moved  under  the 
existing  29  competencies  to  align  with  the  SPRDE  model.  Although  the  SECCM  had  aligned 
with  this  SPRDE  model,  it  was  subsequently  updated  and  changed  significantly  by  the  OSD 
in  2013,  with  further  changes  in  2015. 

Acquisition  Workforce  Career  Fields  SPRDE-SE,  PSE,  and  ENG 

The  pure-and-simple  politics  of  acquisition  education  is  that  many  have 
viewed  it  as  a  deficient  system  that  has  failed  to  make  clear  what  all  students 
need  to  learn  and  whether,  in  fact,  they  have  learned  it.  . . .  The  most 
fundamental  problem  is  our  schools  are  accountable  only  for  educational 
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processes,  not  educational  outcomes. 

Professor  Robert  L.  Hawkings,  Director  of  Curriculum  Development,  DAU 

In  June  2013,  the  SPRDE  Career  Field  Competency  Model  was  refreshed  by  an 
OSD-led  working  group.  The  resulting  model  consists  of  41  competencies  partitioned  into 
four  units  of  competence.  The  SECCM  was  updated  by  remapping  the  existing  KSAs  from 
the  29  SPRDE  competencies  to  the  new  41  competencies.  The  new  SPRDE  model  was 
further  decomposed  into  84  sub-competencies  aligned  to  the  competencies,  with  the  41  high 
level  competencies  shown  in  Table  1. 


Table  1.  The  2013  Refresh  DAU  ENG  Competencies 


ENG  Career  Field  Competency  Model’s  Units  of  Competence 

Competency  Number 

Unit  of  Competence:  Analytical 

1.0 

Mission  Level  Assessment 

20 

stakeholder  Requirements  Definition 

30 

Requirements  Analysis 

4.0 

Architecture  Design 

5,0 

Implementation 

6.0 

Integration 

7.0 

Verification 

8,0 

Validation 

90 

Transition 

10  0 

Design  Considerations 

11  0 

Toots  and  Techniques 

Unit  of  Competence:  Technical  Management 

12  0 

Decision  Analysis 

13  0 

Technical  Planning 

14  0 

Technical  Assessment 

15  0 

Configuration  Management 

16  0 

Requirements  Management 

17  0 

Risk  Management 

18  0 

Data  Management 

19  0 

Interface  Management 

20  0 

Software  Engineering 

21  0 

Acquisition 

Unit  of  Competence:  Professional 

22  □ 

Problem  Solving 

23  0 

Strategic  Ttiinking 

24  0 

professional  Ethics 

25  0 

Leading  H»gh*Perform3nce  Teams 

26  0 

Communication 

270 

Coaching  and  Mentoring 

28  0 

Managing  Stakeholders 

29  0 

Mission  and  Results  Focus 

30  0 

Personal  Effectiveness/Peer  Interaction 

31  0 

Sound  Judgment 

Unit  of  Competence:  Business  Acumen 

32  0 

Industry  Landscapes 

33-0 

Organization 

34  0 

Cost,  Pricing,  and  Rates 

35  0 

Cost  Estimating 

36  0 

Financial  Reporting  and  Metrics 

37  0 

Business  strategy 

38  0 

Capture  Planning  and  Proposal  Process 

39  0 

Supplier  Management 

40  0 

Industry^  Motivation.  Incentives.  Rewards 

41  0 

Negotiations 
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The  competency  changes  between  the  original  SPRDE  and  the  June  2013  refresh 
are  outlined  in  Table  2. 


Table  2.  Summary  of  Competency  Changes  Between  the  SPRDE  Career  Field 

and  ENG  Career  Field  Competency  Models 


Summary  of  Competency  Changes  between  the  SPRDE  Career  Field  and 

ENG  Career  Field  Competency  Models 

Added 

Competencies 

Removed 

Comp«tencies 

Re-phrased  Competencies 

OSD  SPRDE  (Original) 

OSD  ENG  (Update) 

Mission  Level 
Assessment 

Safety  Assurance 

Technical  Data 
Management 

Data  Management 

Cost.  Pricing  & 

Rates 

System  Assurance 

Systems  Engineering 
Leadership 

Leading  High- 
Performance  Teams 

Design 

Considerations 

Reliability.  Availability  & 
Maintainability  (RAM) 

Tools  and 

Techniques 

Technical  Basis  for 

Cost 

Data  Management 

Modeling  and 

Simulation 

Leading  High- 
Performance  Teams 

System  of  Systems 

Coaching  and 
Mentoring 

Managing 

Stakeholders 

Mission  and  Results 
Focus 

Personal 

Effectiveness/Peer 

Interaction 

Sound  Judgment 
Cost  Estimating 

Industry  Landscapes 
Organization 

Financial  Reporting 
and  Metrics 

Business  Strategy 
Capture  Planning 
and  Proposal 

Process 

Supplier 

Management 

Industry  Motivation. 
Incentives,  Rewards 
Negotiations 

While  DAU  designed  and  implemented  their  training  programs  that  satisfy  DAWIA 
requirements,  the  outcomes  of  those  training  programs  are  not  necessarily  targeted  to  the 
general,  holistic  university-level  education  of  systems  engineers  for  the  SE  workforce 
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(Alexander,  2013).  The  overarching  problem  is  whether  or  not  systems  engineering 
competencies  are  being  sufficiently  developed  in  the  SE  workforce. 

In  October  2013,  Under  Secretary  of  Defense  (AT&L)  Frank  Kendall  III  authorized 
the  sunsetting  of  the  SPRDE-PSE  career  field  and  transitioned  all  engineers  in  the  SPRDE- 
PSE  and  SPRDE-SE  career  paths  to  a  consolidated  acquisition  workforce  career  field  titled 
“Engineering”  (Kendall,  2013). 

The  DAU  certification  requirements  for  the  new  ENG  career  field  are  somewhat  more 
substantial  than  the  previous  two  SPRDE  career  fields,  adding  a  few  additional  classes  to 
the  core  requirements.  A  comparison  of  the  core  course  requirements  for  2013  SPRDE-SE 
Levels  I,  II,  and  III  and  2015  ENG  Levels  I,  II,  and  III  are  shown  in  Table  3.  The  old  SPRDE- 
PSE  career  field  certification  required  courses  are  shown  in  addition  to  SPRDE-SE 
requirements.  There  is  no  inclusion  of  the  specific  SPRDE-PSE  requirements  into  the  new 
ENG  requirements. 

Table  3.  Course  Requirements  for  DAU  Certification 


2015  ENG 

ENG 
LEVEL  1 

ACQ101 

Fundamentals  of  Sy sterns  Acquisition 
Management 

SYS  101 

Fundamentals  of  Systems  Planning, 
Research,  Development  and  Engineenng 

CLM017 

Risk  Management 

OLE  101 

Vatue  Engineenng 

'  CLE  P04_ 

Intro  to  Lean  Enterprise  Qxicepts 

ENG 

LEVEL  2 

ACQ  202 

Intermediate  Systems  Acquisition,  Part  A 

ACQ  203 

Inteimediate  Systems  Acquisition,  Part  0 

SYS  202 

Inteimediate  SPRDE,  Part  1 

SYS  203 

Intermediate  SPRDE,  Part  (1 

CLE  003 

Technical  Reviews 

LOG  103 

Reliability,  Avaitabriity  and  Maintainability 

ENG 
LEVEL  3 

SYS  302 

NOT  REQUIRED 

CLL  008 

Design  for  the  supportability  in  DoD 
Systems 

CLE  068 

1 

Iies4gn0)9iar^$i]|^^ 

Systems 

DoD 

CLE  012 

1 

046301 

Open  Sy  stents  Arehitectufe 

Leadership  in  Engmeenng  Defense 

Syslen» 

2013  SPROE‘SE 

SE 

LEVEL  1 

Course 

ACQ  101 

Fuixla  menials  of  Systems 
Acquisition  Management 

SYS  101 

Furlda  mentals  of  Systems 
Planning,  Research, 

Development  and 

Engineering 

CLM  017 

Risk  Management 

SE 

LEVEL  2 

ACQ102A 

tntermedfate  Systems 
Acquisition,  Part  A 

ACQ102B 

Intermedfate  Systems 
Acquisition,  Part  8 

SYS  202 

Intermediate  SPRDE,  Part  1 

SYS  203 

Intermediate  SPRDE,  Part 

11 

CLE  003 

Technical  Reviews 

SE 

LEVEL  3 

SYS  302 

Technical  Leadership  in 
Systems  Engineering 

CLL  008 

Design  for  the  supportability 
in  DoO  Systems 
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In  addition  to  the  development  of  a  new  DAU  curriculum  to  match  the  new  ENG 
career  field,  the  former  SPRDE  model  was  renamed  as  the  ENG  Career  Field  Competency 
Model  in  the  same  action  in  October  promulgated  by  the  OSD.  At  the  time,  no  changes  were 
made  to  the  June  2013  refresh  SPRDE  model  contents.  In  February  2015,  however,  a  new 
OSD  WG  was  formed  to  refresh  the  ENG  Career  Field  Competency  Model.  The  transition 
from  the  original  pre-2013  SPRDE  Career  Field  Competency  Model  to  the  current  ENG 
Career  Field  Competency  Model  is  summarized  graphically  in  Figure  3.  This  2015  model 
update  is  still  in  process  as  of  April  2015. 


SPRDE  Career  Field 

2013  ENG  Career  Field  \\ 

2015  ENG  Career  N 

Competency  Model  V 

1  Competency  Model  ^ 

Field  Competency 

•  29  Competencies 

Refresh  \ 

V  Model 

•  45  Sub- 

^  *41  Competencies 

iX  *41  Competencies 

Competencies 

^  •  84  Sub- 

^  •  XX  Sub- 

•  3  Units  of 

n  Competencies 

If  Competencies 

Competence 

N  *4  Units  of 

//  Competence  / 

ff  *4  Units  of 
f  Competence 

Figure  3.  SPRDE  to  ENG  Transition 

Competency  Classification  Verification 

The  current  SECCM  competencies  align  with  the  ENG  Career  Field  Competency 
Model  2015  version.  The  leading  verb  phrases  in  the  statements  of  the  KSA  within  the 
competencies  have  been  redefined  using  key  words  from  Bloom’s  taxonomomic 
classicifcation  schema.  This  naturally  partitioned  them  further  into  specific  cognitive  or 
affective  categories  found  within  Bloom’s  schema.  Specifically,  the  version  of  Bloom’s 
taxonomy  as  adapted  by  Krathwohl  (2002),  shown  in  Figure  4,  was  used  for  the  cognitive 
and  affective  learning  domains. 


Apply 


Understand 


Figure  4.  Bloom’s  Taxonomy  Cognitive  Domain  From  Krathwohl  (2002) 

(MMI,  n.d.) 
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The  significance  of  using  Bloom’s  taxonomic  classification  schema  is  that  in 
research,  how  engineering  students  learn  is  often  categorized  using  Bloom’s  taxonomy.  For 
instance,  engineering  education  involves  learning  objectives,  which  are  typically  organized 
around  Bloom’s  taxonomy  of  cognitive  and  affective  processes.  The  cognitive  domain 
involves  knowledge  and  the  development  of  intellectual  skills,  whereas  the  affective  domain 
deals  with  the  motivations  and  attitudes  involved  in  learning  (Clark,  1999).  Cognitive  and 
affective  processes  within  Bloom’s  taxonomy  refer  to  levels  of  observable  actions  that 
indicate  learning  is  occurring. 

Bloom’s  taxonomy  provides  hierarchical  outcome  categories  or  levels  that  range 
from  simple  to  complex  thought  processes.  Once  the  Bloom’s  level  for  each  KSA  was 
identified,  each  of  the  KSAs  were  mapped  to  one  of  three  notional  career  levels  designated 
as  SE-01  Entry  Level,  SE-02  Journey  Level  or  SE-03  Expert  Level,  as  shown  in  Figure  5. 
The  KSAs  generally  attributed  to  lower  level  Bloom’s  learning  process  were  mapped  into 
SE-1  Entry  Level,  the  KSAs  with  intermediate  level  Bloom’s  learning  processes  were 
mapped  into  SE-2  Journey  Level  and  the  KSAs  with  higher  level  Bloom’s  learning  processes 
were  mapped  into  the  SE-3  Expert  Level.  In  an  effort  to  create  a  foundation  for  SE  career 
development  within  the  DON,  the  KSAs  were  then  mapped  into  competencies  across 
proficiency  levels. 


SECCM  Structure 


GS-ltoS 


i 


GS-9tol2 

GS^lBtolS 

- ^ 

— — ^ 

- ^ 

1 

-L_ 

H _ 

A  1 

-n — 

H  1 

^ _ 1 

■ 

H _ 1 

4 _ 1 

Lj 

Figure  5.  Typical  SECCM  Career  Development  Levels 

The  alignment  of  KSAs  to  competencies  is  just  a  notional  starting  point,  as  the 
research  team  intends  that  any  assignment  of  various  KSAs  to  different  career  development 
levels  can  be  conducted  by  an  organization  in  its  own  specific  implementation  of  the 
SECCM  as  a  career  competency  development  tool. 

A  baseline  review  conducted  by  SMEs  verified  that  the  KSAs  were  aligned  to 
reasonable  competencies.  As  a  result  of  the  WG  effort.  If  a  group  consensus  was  that  a 
KSA  was  not  aligned  to  an  appropriate  competency,  the  KSA  was  re-assigned  to  one 
deemed  more  appropriate  by  the  SMEs.  The  baseline  review  also  identified  some  KSAs  that 
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did  not  belong  in  the  model.  If  the  SMEs  and  stakeholders  on  the  SECCM  team  felt  that  a 
KSA  did  not  apply  to  an  SE  application,  the  KSA  was  eliminated  from  the  model.  In  some 
instances,  SMEs  added  KSAs  to  the  model  based  on  their  experience.  Following  this 
iterative  process,  redundant  KSAs  were  deleted  and  vague  KSAs  were  re-written.  In  an 
effort  to  enforce  consistency  in  the  model  (while  also  properly  using  Bloom’s  taxonomic 
classification  schema),  each  KSA  was  updated  to  have  an  action  verb  at  the  very  beginning 
of  the  sentence.  The  verbs  were  all  converted  to  present  tense  for  consistency  (Whitcomb, 
Khan,  &  White,  2014). 

Compliance  With  Uniform  Guidelines 

Now  that  the  competency  model  for  professionals  performing  systems  engineering 
activities  is  developed,  the  next  step  is  to  finalize  the  model  through  an  occupational 
analysis.  The  Office  of  Personnel  Management’s  (0PM)  Leadership  and  Workforce 
Development  Assessment  (LWDA)  team  has  joined  the  Naval  Postgraduate  School  and 
SMEs^  to  assist  in  the  refinement,  confirmation,  and  strategic  planning  required  to  ensure 
the  systems  engineering  competency  model  is  a  legally  defensible,  relevant,  and  sound  tool 
that  may  be  used  for  a  variety  of  human  resources  purposes  (e.g.,  career  path  modeling, 
skills  gap  assessment,  selection  tool  development).  To  ensure  the  competency  model  will 
be  able  to  serve  as  the  foundation  for  establishing  content  verification  for  future  human 
resources  actions,  0PM  uses  an  approach  to  the  occupational  analysis  in  compliance  with 
the  Uniform  Guidelines.  This  approach  gathers  input  from  SMEs  through  panels  and  an 
occupational  analysis  survey. 

Outline  of  OPM  Occupational  Analysis  Methodology 

Beginning  with  the  model  created  by  NPS,  the  OPM  is  taking  a  four-pronged 
approach  to  the  occupational  analysis:  review  of  occupational  information,  facilitation  of 
SME  panels,  administration  of  surveys,  and  documentation.  The  occupational  analysis 
methodology  focuses  on  identifying  the  competencies  and  tasks  that  are  critical  for 
employees  functioning  as  systems  engineers.  This  method  of  occupational  analysis 
establishes  which  competencies  are  suitable  for  assessment  in  human  resources  activities. 

The  OPM  began  the  occupational  analysis  with  a  review  of  the  SECCM  along  with 
additional  occupational  information  provided  by  NPS  and  other  DoD  components,  including 
the  MDA.  The  occupational  information  served  to  further  define  the  competencies  included 
in  the  existing  NPS  model.  Adding  descriptions  to  the  competencies  serves  to  ensure  each 
competency  included  in  the  model  is  clear  and  unique.  The  OPM  also  conducted  an  initial 
review  of  the  KSAs  included  in  the  SECCM  to  refine  the  list.  LWDA  personnel  research 
psychologists  removed  or  revised  KSAs  that  were  not  behaviorally  based  or  measurable 
characteristics  to  ensure  the  resulting  task  statements  had  the  characteristics  necessary  to 
support  a  variety  of  human  resources  activities. 

LWDA  personnel  research  psychologists  then  facilitated  SME  panels  to  further  refine 
the  SECCM.  Panels  were  held  first  with  incumbents  who  currently  perform  systems 
engineering  activities  and  then  with  individuals  who  supervise  those  who  perform  systems 
engineering  activities.  NPS  recruited  SMEs  to  participate  in  the  panels,  requiring  SMEs  to 


'  SME  teams  consisted  of  incumbents  and  supervisors  aeross  the  Department  of  Defense  (DoD)  who  either  eurrently  engage 
in  or  supervise  systems  engineering  aetivities. 
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meet  experience  criteria  to  ensure  each  SME  had  a  minimum  level  of  familiarity  with 
systems  engineering  activities.  SMEs  provided  input  to  further  revise  competency  definitions 
and  task  statements,  identify  competencies  and  tasks  critical  to  systems  engineering  which 
were  not  represented  in  the  existing  model,  and  eliminate  tasks  not  representative  of  the 
job. 

The  revised  competencies  and  tasks  will  serve  as  the  foundation  for  an  occupational 
analysis  survey.  The  0PM  will  invite  employees  who  perform  systems  engineering  activities 
and  their  supervisors  to  participate  in  the  survey,  only  retaining  data  from  employees  with 
minimum  experience  levels  to  ensure  adequate  familiarity  with  systems  engineering  work. 
The  0PM  will  further  ensure  that  participants  in  the  survey  have  relevant  experience  by 
asking  NPS  to  identify  the  population  and  by  confirming  each  participant’s  experience 
through  survey  branching  methodology.  The  survey  branching  will  require  participants  to 
respond  to  questions  designed  to  distinguish  participants  who  function  as  a  systems 
engineer  from  those  who  serve  in  other  engineering  disciplines.  Survey  participants  will 
evaluate  each  competency  and  task  in  the  SECCM  on  criteria  such  as  frequency, 
importance,  required  immediately  upon  entry  into  the  position,  and  need  for  training. 

LWDA  personnel  research  psychologists  will  conduct  statistical  analyses  on  the 
survey  data  to  determine  which  competencies  and  tasks  are  critical  to  the  successful 
performance  of  systems  engineering  activities.  To  identify  the  critical  tasks,  the  research 
psychologists  will  analyze  task  ratings  of  frequency  and  importance.  Competencies  critical 
for  performing  systems  engineering  activities  will  be  identified  by  analyzing  competency 
ratings  of  importance  and  need  at  entry.  The  resulting  critical  tasks  and  competencies  will 
make  up  the  occupational  profile  for  individuals  performing  systems  engineering  work. 

In  conformance  with  legal  and  professional  guidelines,  the  0PM  will  document  the 
methodology  and  results  for  all  phases  of  the  occupational  analysis.  The  documentation  is  a 
necessary  component  for  demonstrating  that  the  process  is  sufficient  to  serve  as  a 
component  of  a  content  validation  approach^  for  ensuring  the  validity  of  future  human 
resources  activities. 

Identifying  SE  Population 

Identifying  the  population  of  systems  engineers  in  any  organization  is  currently  a 
challenge  faced  by  the  DoN  and  other  defense  organizations.  The  SE  population  is  needed 
to  identify  those  SE  to  include  in  the  survey  pool.  The  SE  population  is  identified  based  on 
input  from  all  participating  organizations.  There  is  no  single  best  way  to  identify  a  systems 
engineer,  so  each  organization  must  attempt  to  identify  their  own  population  based  on 
identifying  engineers  who  perform  tasks  related  to  SE.  The  population  is  required  to 
complete  the  cost  estimate  document  that  is  required  to  obtain  DoD  Survey  approval. 
Approval  is  in  process  and  will  be  obtained  prior  to  deploying  the  survey. 


^  “Evidence  of  the  validity  of  a  test  or  other  selection  procedure  by  a  content  validity  study  should 
consist  of  data  showing  that  the  content  of  the  selection  procedure  is  representative  of  important 
aspects  of  performance  on  the  job  for  which  the  candidates  are  to  be  evaluated”  (Biddle  Consulting 
Group,  2013). 
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Relationship  to  DCAT 

The  Defense  Civilian  Personnel  Advisory  Service  (DCPAS),  Strategic  Human  Capital 
Planning  Division  (SHCPD)  is  responsible  for  the  implementation  of  the  DoD  competency- 
based  approach  for  workforce  pianning.  This  approach  inciudes  competency  model 
development  for  mission  criticai  occupations  and  all  major  occupational  series,  assessment 
of  civiiian  workforce  competency  gaps,  and  identification  of  civilian  workforce  competencies 
needed  now  and  into  the  future.  This  initiative  supports  the  DoD’s  abiiity  to  meet  the 
iegisiative  requirements  estabiished  in  10.U.S.C  115b.  The  DoD  designed,  deveioped,  and 
depioyed  the  Defense  Competency  Assessment  Tool  (DCAT)  as  a  Department-wide,  online 
tool  to  validate  occupationai  competency  models  and  assess  civilian  empioyees’  proficiency 
ieveis  and  competency  gaps. 

Empioyees  are  seiected  to  participate  in  DCAT  through  a  stratified  random  sampie 
process  based  on  civilian  employee  grade  and  particular  occupational  series  being 
assessed.  DCAT  interfaces  with  the  Defense  Civiiian  Personnei  Data  System  (DCPDS);  this 
feature  iimits  DCAT  use  to  occupations  ciassified  under  the  U.S.  Standard  Occupational 
Classification  System. 

DCAT  assesses  civiiian  empioyees  by  their  individuai  series’  competency  model,  not 
by  functional  competency  models.  Since  the  System  Engineering  discipiine  is  a  functional 
career  field  that  cuts  across  multiple  occupational  series,  currently  DCAT  does  not  have  the 
capability  to  assess  the  SECCM,  so  the  0PM  was  brought  in  to  the  SECCM  team  to 
accomplish  the  verification  process. 

SECCM  Implementation 

The  SECCM  is  implemented  in  a  spreadsheet  format  to  make  it  easy  to  manipulate 
based  on  any  desired  outcome  for  customization  of  a  competency  model  within  any  defense 
organization.  The  goai  is  that  an  organization  can  use  the  model  as  a  foundation  and  have 
an  opportunity  to  customize  it  to  meet  the  needs  of  the  organization.  The  SECCM  is  a  great 
tool  to  use  because  a  baseline  review  of  the  model  by  organizations  across  the  DoD  has 
been  completed.  The  baseline  review  was  compieted  to  certify  that  the  KSAs  were  ciearly 
defined  and  applicable.  As  previously  mentioned,  the  41  competencies  in  the  SECCM  mirror 
the  competencies  identified  in  the  ENG  Career  Field  Competency  Model.  The  KSAs  in  the 
model  were  taken  from  several  other  successful  existing  competency  modeis.  These  KSAs 
were  re-written,  categorized  into  biooms  ieveis  of  cognitive  and  affective  domains,  and 
organized  prior  to  adopting  them  into  the  SECCM.  One  organization,  the  Space  and  Navai 
Systems  Center  (SSC)  Atlantic,  has  implemented  their  SE  competency  modei  using  the 
SECCM.  In  addition,  the  NPS  SECCM  project  team  is  coliaborating  as  members  of  the 
International  Council  on  Systems  Engineering  (INCOSE)  Competency  Working  Group 
(CWG)  to  develop  an  INCOSE  SE  competency  model. 

SPAWAR  Uses  SECCM 

SSC  Atlantic  was  recently  challenged  to  establish  a  competency  development  model 
for  systems  engineers.  As  of  March  2015,  SSC  Atiantic  consists  of  close  to  4,000  U.S. 
government  employees — approximately  900  of  which  can  be  considered  systems  engineers. 
Over  240  integrated  product  teams  (IPTs)  in  SSC  Atiantic  work  to  deliver  various  IT-related 
end  item  products  to  Navai,  Joint  and  Coalition  warfighting  customers.  The  range  of 
engineering  processes,  technologies,  missions,  and  customers  supported  by  the  SSC 
Atiantic  engineering  department  covers  a  wide  spectrum.  In  order  to  better  understand  the 
KSAs  required  for  a  systems  center,  KSAs  were  organized  into  competency  dimensions — 
process,  technoiogy,  and  mission  (excluding  leadership  skiiis  for  the  sake  of  this  paper  as 
they  apply  to  everyone — not  just  systems  engineers).  Having  a  robust  set  of  competency 
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areas  for  each  of  these  dimensions  helps  ensure  that  systems  engineers  are  well  rounded  in 
order  to  provide  technical  leadership  to  multi-disciplinary  teams  with  role-diverse  team 
members  (Walter,  2013). 

Within  each  dimension,  competency  areas  were  defined  and  prioritized  based  on  a 
standard  set  of  SE  use  cases  most  commonly  experienced  at  SSC  Atlantic,  as  well  as 
based  on  DoD  and  industry  standards.  High  prioritization  of  competency  areas  associated 
with  requirements,  architecture  design,  software  engineering,  and  system  assurance 
highlights  the  importance  for  sound  up-front  systems  engineering  process  execution,  IT 
systems’  increasing  reliance  on  software,  and  the  paramount  need  for  cybersecurity.  By 
defining  subroles  (or  specialty  areas)  for  a  systems  engineer,  further  KSAs  were  defined 
that  stress  certain  competency  areas  over  others.  Identifying  relevant  and  authoritative 
competency  areas  and  KSA  sources  for  a  competency  framework  is  critical,  as  there  is  no 
need  to  recreate  data  that  has  already  been  adequately  developed  by  several  other  relevant 
and  established  industry  and  DoD  organizations.  Competency  areas  and  KSAs  from  the 
Naval  SECCM  were  heavily  used  to  populate  the  SSC  Atlantic  systems  engineer 
competency  development  model — particularly  in  the  SE  process  dimension.  For  the 
purposes  of  developing  an  SE  competency  framework  for  SSC  Atlantic,  the  SECCM 
provides  a  wide  array  of  KSAs  from  which  to  choose,  along  with  recommended  competency 
development  model  (CDM)  stages  (levels)  for  each  individual  knowledge,  skill,  or  ability. 

Due  to  SSC  Atlantic’s  mission  focus  on  IT  and  cyberspace,  the  NIST  national  cybersecurity 
workforce  framework  also  proved  highly  useful  in  tailoring  an  SE  competency  framework. 
Several  other  competency  frameworks  and  sources  were  used  to  a  lesser  degree  to 
populate  the  underlying  KSA  database  used  for  roles  all  across  SSC  Atlantic. 

In  order  to  establish  a  complete  set  of  KSAs  at  each  competency  development 
stage,  a  layered-cake  approach  was  taken,  meaning  that  KSAs  are  applicable  to  an 
increasingly  narrowed  sector  of  individuals.  For  example.  Figure  6  shows  how  every  SSC 
Atlantic  employee  needs  leadership  skills,  members  of  the  entire  engineering  department 
(a.k.a.  “competency”)  require  “core”  engineering  skills,  all  systems  engineers  require  a 
certain  set  of  KSAs,  and  then  specific  sub-roles  or  types  of  systems  engineers  require  yet  a 
separate  set  of  KSAs.  Each  of  these  KSA  sets  is  ultimately  part  of  a  systems  engineer’s 
competency  development  model.  In  order  to  establish  systems  engineering  roles  that  could 
be  well  understood  across  the  organization,  we  also  examined  the  roles  that  interact  with  a 
systems  engineer  in  order  to  determine  where  KSAs  should  be  shared  across  the  roles  or 
unique  to  one  or  the  other. 


Figure  6.  SSC  Atlantic  SE  Competency  Framework  KSA  Pyramid 

Analysis  was  conducted  to  understand  how  these  KSAs  can  and  should  be  obtained. 
The  most  common  methods  for  an  individual  to  obtain  a  KSA  are  through  educational 
training  (DAU,  degrees,  or  certifications),  in-house-developed  training  courses/workshops. 
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and  on-the-job  training  (OJT).  DAU  engineering  classes  can  be  effective  when  providing 
systems  engineers  with  basic  knowledge  and  comprehension  of  the  SE  life  cycle 
processes — particularly  in  the  areas  of  acquisition  and  risk  management.  OJT  can  be 
enhanced  when  coupled  with  targeted  rotational  opportunities  and  job  shadowing 
opportunities.  If  approached  systematically,  immeasurable  value  can  be  obtained  from 
developing  in-house  SE  training  that  engages  systems  engineers  at  all  levels  of  the 
workforce.  When  assessing  the  competency  of  systems  engineers,  care  was  taken  to 
choose  an  assessment  process  and  associated  assessment  methodologies  that  were 
relatively  thorough  yet  not  overly  cumbersome,  time-consuming,  and  costly. 

From  the  perspective  of  an  individual  desiring  to  develop  over  the  course  of  their 
career  in  the  role  of  a  systems  engineer,  they  are  provided  with  guidance  on  how  to  do  so. 
Figure  7  illustrates  how  SSC  Atlantic  systems  engineers  seek  to  better  understand  their  role 
and  CDM;  complete  a  CDM  self-assessment  package  where  they  address  each  required 
KSA;  review  feedback  from  a  competency  assessment  panel  regarding  their  KSA  gaps; 
complete  an  individual  development  plan  which  defines  what  developmental  activities  and 
training  they  plan  to  take  in  order  to  satisfy  each  KSA;  complete  their  training  and/or 
developmental  activities;  update  their  I  DP;  and  then  submit  for  reassessment  against  the 
CDM.  From  the  perspective  of  a  competency  manager  or  the  training  department,  this 
provides  a  much  more  streamlined  method  for  determining  the  required  training  for  SSC 
Atlantic,  as  training  and  developmental  activities  can  be  targeted  directly  at  the  KSA  gaps  of 
the  workforce. 


Figure  7.  SSC  Atlantic  SE  Competency  Model  Development 

Relationship  to  INCOSE  Competency  Model  Development 

The  NPS  SECCM  project  team  is  collaborating  with  the  INCOSE  CWG  to  develop  a 
new  INCOSE  SE  Competency  Model.  An  Initial  evaluation  of  the  transfer  of  SECCM  KSAs 
into  the  INCOSE  SE  Competency  model  (draft  version),  reveals  that  almost  half  (49%)  of 
the  SECCM  KSAs  are  in  the  lowest  level  of  cognitive  and  affective  learning  domains  of 
Remember,  Understand,  Receive  Phenomena  &  Respond  to  Phenomena.  Overall,  the 
majority  (66%)  of  the  SECCM  KSAs,  as  aligned  with  the  INCOSE  competencies,  are 
mapped  to  the  cognitive  domain.  The  remaining  (34%)  KSAs  are  mapped  into  the  affective 
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domain.  Figures  8  and  9  illustrate  the  percentage  of  cognitive  and  affective  KSAs  as 
captured  in  the  SECCM,  by  learning  domain. 


Figure  8.  Count  and  Percentage  of  Cognitive  KSAs  Within  SECCM 

Cognitive  Domain 
Total  KSAs:  1,732 


Figure  9.  Count  and  Percentage  of  Affective  KSAs  Within  SECCM 

Affective  Domain 
Total  KSAs:  869 

Characterize 
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In  assessing  the  draft  results  of  the  transfer  of  SECCM  KSAs  into  the  INCOSE  SE 
competencies,  it  can  be  seen  that  SEs  need  to  be  highly  competent  in  KSAs  related  to  the 
cognitive  domains  of  Concurrent  Engineering,  Design  Considerations/Specialty  Engineering, 
Lifecycie  Process  Definitions  and  Technicai  Assessment  and  Controi.  Similarly, 
observations  regarding  the  results  of  the  affective  domain  processes  show  that  Leadership 
and  Team  Dynamics  are  important  requisite  Systems  Engineering  competencies.  These 
findings  are  detailed  in  Table  4. 


Figure  10.  Transfer  of  SECCM  KSAs  into  INCOSE  competencies  (draft):  Count  of 

Cognitive  and  Affective  KSAs 


INCOSE  Competency 

Count  of  KSAs 

Cognitive 

Affective 

Acquisition 

11 

Architecture  Definition 

30 

2 

Communication 

31 

66 

Concurrent  Engineering 

128 

57 

Configuration  Management 

61 

21 

Critical  Thinking 

98 

4C 

Decision  Management 

76 

24 

Design  Considerations/Specialty  Engineering 

142 

64 

Engineering  and  Scientific  Fundamentals 

12 

Enterprise  Integration 

1 

Ethics 

17 

24 

Implementation 

19 

22 

Information  Management 

13 

Integration 

18 

6 

Integration 

20 

16 

Integration  of  Specialisms 

3 

Interface  Management 

58 

11 

Leadership 

78 

157 

Lifecycle  Process  Definition 

308 

42 

Mathematics/Logic/Quantitative  Analysis 

1 

Modeling  &  Simulation 

62 

22 

Negotiation 

3 

Reliability,  Maintainability  &  Availability  Analysis 

6 

1 

Safety  Analysis 

2 

System  Requirements  Definition 

63 

2£ 

Team  Dynamics 

29 

12C 

Technical  Assessment  &  Control 

121 

45 

Technical  Planning 

64 

5 

Technical  Risk  Management 

74 

27 

Transition 

48 

£ 

Validation 

40 

2£ 

Verification 

95 

3C 

Total 

1,732 

86S 

Grand  Total  (Cognitive  &  Affective) 

2,601 
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Summary 

Progress  on  the  SECCM  has  positioned  the  DOD  as  a  leader  in  the  human 
resources  management  aspect  through  this  systems  engineering  competency  modeling 
effort.  The  SECCM  identifies  a  collection  of  KSAs  that  define  the  basis  for  developing 
effective  systems  engineers.  The  NPS  research  team  contends  that  along  with  its 
applicability  as  a  valuable  resource  as  a  model  that  can  be  used  for  key  HR  functions  the 
SECCM  can  also  be  used  to  assist  undergraduate  and  graduate  academic  programs  in 
specifying  student  outcomes  and  learning  objectives  within  systems  engineering  programs  - 
to  ensure  students  have  the  entry-level  KSAs  required  to  perform  successfully  once  in  the 
field.  The  implications  of  the  research  can  also  be  used  to  develop  structured  curriculum 
content,  assessment,  and  continuous  process  improvement  techniques  related  to  the 
development  of  SE  learning,  and  to  develop  more  valid  and  reliable  instruments  for 
assessing  what  systems  engineers  need  to  learn,  need  to  know,  and  need  to  do. 

References 

Alexander,  J.  (2013).  Analysis  of  training  requirements  and  competencies  for  the  naval 
acquisition  systems  engineering  workforce  (Master’s  thesis.  Naval  Postgraduate 
School).  Retrieved  from  http://hdl. handle. net/1 0945/34618 

Biddle  Consulting  Group.  (2013).  Uniform  guidelines  on  employee  selection  procedures. 
Retrieved  from  http://uniformquidelines.eom/uniformquidelines.html#22 

Clark,  D.  (1999).  Bloom’s  taxonomy  of  learning  domains.  Retrieved  from 
http://www.nwlink.com/~donclark/hrd/bloom.html 

Delgado,  J.  (2014).  Director,  Acquisition  Career  Management  (DACM)  newsletter.  Retrieved 
from 

http://www.secnav.navv.mil/rda/workforce/Documents/DACM%20Corner%20Newsletter 

%20-%200ctober%202014.pdf 

Holt,  J.,  &  Perry,  S.  A.  (2011).  A  pragmatic  guide  to  competency.  Swindon,  United  Kingdom: 
British  Informatics  Society. 

Joshi,  K.  D.,  Kvasny,  L.,  McPherson,  S.,  Trauth,  E.,  Kulturel-Konak,  S.,  &  Mahar,  J.  (2010). 
Choosing  IT  as  a  career:  Exploring  the  role  of  self-efficacy  and  perceived  importance  of 
IT  skills.  Presented  at  the  International  Conference  on  Information  Systems,  Saint 
Louis,  MO. 

Kendall,  F.  (2013,  September  6).  Sunsetting  of  the  systems  planning,  research, 

development,  and  engineering — program  systems  engineer  acquisition  workforce 
career  path  and  the  renaming  of  the  systems  planning,  research,  development,  and 
engineering  acquisition  workforce  career  field  [Memorandum].  Washington,  DC:  DoD. 

Khan,  R.  (2014).  Assessment  of  self-efficacy  in  systems  engineering  as  an  indicator  of 

competency  level  achievement  (Master’s  thesis.  Naval  Postgraduate  School).  Retrieved 
from  http://hdl.handle.net/10945/42660 

LaRocca,  M.  (n.d.).  Career  and  competency  pathing:  The  competency  modeling  approach. 
Retrieved  October  21 , 2013  from 

http://edweb.sdsu.edu/people/arossett/pie/interventions/career  1.htm 
MMI.  (n.d.).  Taxonomies  of  learning:  Visual  comparisons  of  the  taxonomies.  Retrieved  from 
http://www.mmiweb.orq.uk/downloads/bloom.html 

Walter,  P.  G.  (2013).  A  model  for  effective  systems  engienering  workforce  development  at 
Space  and  Naval  Warfare  Systems  Center  (SCC)  Atlantic  (Master’s  thesis.  Naval 
Postgraduate  School).  Retrieved  from  http://hdl.handle.net/10945/37738 


-287- 


ACQUISITION  RESEARCH  PROGRAM: 
CREATING  SYNERGY  EORINEORMED  CHANGE 


Whitcomb,  C.,  Khan,  R.,  &  White,  C.  (2014).  Systems  engineering  competency  report. 
Retrieved  from  https://calhoun.nps.edu/handle/10945/43424 

White,  C.  (2014).  Development  of  systems  engineering  competency  career  development 
model:  An  analytical  approach  using  Bloom’s  taxonomy  (Master’s  thesis.  Naval 
Postgraduate  School).  Retrieved  from  http://hdl.handle.net/10945/42752 


ACQUISITION  RESEARCH  PROGRAM: 
CREATING  SYNERGY  EOR  INFORMED  CHANGE 


-288- 


ACQUISITION  RESEARCH  PROGRAM 

GRADUATE  SCHOOL  OE  BUSINESS  &  PUBLIC  POLICY 

NAVAL  POSTGRADUATE  SCHOOL 

555  DYER  ROAD,  INGERSOLL  HALL 

MONTEREY,  CA  93943 


www.acquisitionresearch.net 


